{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Object detection on PYNQ\n",
    "\n",
    "This notebook demonstrates the typical working flow of our solution. In our solution, a shared library `libssd.so` is first created from C/C++ sources and NN models, with the help of [DNNDK](http://www.deephi.com/dnndk.html). `libssd.so` exports necessary handlers to initialize, operate and terminate the object detection IP (DPU) running on the fabric. Then, the python notebook accesses the shared library and its exposed handlers to interact with the fabric. Most scheduling work is done within the shared library. For details, refer to the C++ codes.\n",
    "\n",
    "Either to start from scratch (compiling from source codes and models to the final binary file) or just have a try of our solution (using our binaries), there are some prerequisites:\n",
    "- Your PYNQ board with a tuned linux kernel\n",
    "- OpenCV (C++ version)\n",
    "- Deep Neural Network Development Kit (DNNDK) from DeePhi\n",
    "In the `prerequisites` folder we have provided the device tree, the DNNDK package along with instructions on installation. If you have problems configuring DNNDK, please contact [DeePhi](http://www.deephi.com).\n",
    "\n",
    "Below we illustrate our solution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialization\n",
    "\n",
    "Here we import the necessary packages and set up the environment. The shared library in Python is accessed using [cffi](https://cffi.readthedocs.io/). The simplest mode (ABI, in-line) already satisfies our need. First, exported interfaces are declared again in Python; then the shared library is opened. With `cffi.new` we are free to pass the arguments and call the functions in the shared library now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import math\n",
    "import os\n",
    "import time\n",
    "from datetime import datetime\n",
    "from pynq import Overlay\n",
    "from preprocessing import *\n",
    "from iou import *\n",
    "from cffi import FFI\n",
    "\n",
    "team = 'TGIIF'\n",
    "agent = Agent(team)\n",
    "print(\"Team created\")\n",
    "\n",
    "ffi = FFI()\n",
    "ffi.cdef('''\n",
    "typedef struct {\n",
    "    int label;\n",
    "    int xmin;\n",
    "    int xmax;\n",
    "    int ymin;\n",
    "    int ymax;\n",
    "    float confidence;\n",
    "} result_t;\n",
    "''')\n",
    "\n",
    "ffi.cdef('''\n",
    "void dpu_initialize(char *lib_path);\n",
    "result_t dpu_detect_single(char *path);\n",
    "void dpu_detect_list(char *, unsigned);\n",
    "void dpu_clear(void);\n",
    "void dpu_destroy(void);\n",
    "result_t *dpu_get_results(void);\n",
    "''')\n",
    "\n",
    "lib_path = os.path.join(os.getcwd(), \"libraries/libssd.so\")\n",
    "dpu_lib = ffi.dlopen(lib_path)\n",
    "\n",
    "c_lib_path = ffi.new(\"char []\", lib_path.encode())\n",
    "print(\"Lib opened:\", lib_path)\n",
    "\n",
    "c_results = ffi.new(\"result_t *\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overlay loading\n",
    "\n",
    "The bitstream file is loaded to the PL side of PYNQ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "OVERLAY_PATH = os.path.join(OVERLAY_DIR, \"TGIIF/pynq_dpu_142m.bit\")\n",
    "overlay = Overlay(OVERLAY_PATH)\n",
    "print(\"Overlay loaded: {}\".format(OVERLAY_PATH))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Image processing\n",
    "\n",
    "This step we process the images using the declared functions. We need to all `dpu_initialize` to get DPU ready. To obtain the best performance, `dpu_detect_list` is called for processing images. The API accepts the name of a text file which contains the list of image paths as the argument and returns an array of results. To detect images one by one, you could use `dpu_detect_single`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "interval_time = 0\n",
    "total_time = 0\n",
    "total_num_img = len(agent.img_list)\n",
    "result = list()\n",
    "agent.reset_batch_count()\n",
    "\n",
    "# Initialize DPU\n",
    "dpu_lib.dpu_initialize(c_lib_path)\n",
    "print(\"DPU initialized\")\n",
    "\n",
    "# Start processing\n",
    "result_records = []\n",
    "for i in range(math.ceil(total_num_img/BATCH_SIZE)):\n",
    "    # get a batch from agent\n",
    "    batch = agent.send(interval_time, agent.img_batch)\n",
    "    \n",
    "    # timer starts\n",
    "    start = time.time()\n",
    "    with open(agent.coord_team + \"/imgs.txt\", 'w') as fimg:\n",
    "        fimg.write(IMG_DIR+\"\\n\")\n",
    "        for img in batch:\n",
    "            fimg.write(img+'\\n')\n",
    "    print(\"Image list created\")\n",
    "    \n",
    "    c_imgs_file = ffi.new(\"char []\", (agent.coord_team+\"/imgs.txt\").encode())\n",
    "    c_img_num = ffi.new(\"unsigned *\")\n",
    "    c_img_num[0] = BATCH_SIZE\n",
    "    dpu_lib.dpu_detect_list(c_imgs_file, c_img_num[0])\n",
    "    c_results = dpu_lib.dpu_get_results()\n",
    "    print(\"Current batch processed\")\n",
    "    \n",
    "    result_records += [[c_results[j].xmin, c_results[j].xmax, c_results[j].ymin, c_results[j].ymax,\n",
    "                        c_results[j].confidence, c_results[j].label] for j in range(c_img_num[0])]\n",
    "        \n",
    "    # timer stop after PS has received image\n",
    "    end = time.time()\n",
    "    t = end - start\n",
    "    print('Processing time: {} seconds.'.format(t))\n",
    "    total_time += t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results storing\n",
    "\n",
    "Detection results are stored into xml files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write misc info\n",
    "agent.write(total_time, total_num_img, team)\n",
    "\n",
    "# Write detection results into xml files\n",
    "agent.save_results_xml(result_records)\n",
    "print(\"XML results written successfully.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleaning up\n",
    "\n",
    "`dpu_destroy` is called to release the system resources and make DPU idle. To start DPU again, you can call `dpu_intialize` later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dpu_lib.dpu_destroy()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
